IN THE CLAIMS 



1 . (Currently amended) A method for providing collateral information for 
inclusion with an information stream, comprising steps of: 



examining the information stream to recognize a presence of events that 
occur in the information stream , wherein said events are derived from the 
information stream based on one or more predetermined taxonomies ; 



events; and 

analyzing results of said database queries so as to rank and select said 
results to be inserted into the information stream as information that is collateral 



2. (Original) A method as in claim 1 , wherein the step of analyzing 
comprises a step of ranking the database query results based on a plurality of 
criteria. 

3. (Currently amended) A method as in claim 2, wherein the plurality of 
criteria comprise a score derived from a free text search of the database using 
text that is automatically extracted from the information stream, on a number of 
named entities appearing in the text and in the database query results, and on a 
taxonomy path score, where the taxonomy path score represents an amount of 
relatedness between a taxonomy-related information element found in the text 
and a tree of the predetermined taxonomy tr ee taxonomies . 

4. (Currently amended) A method as in claim 1 , wherein the database 
queries are automatically generated based on information corresponding to a list 
that identifies topics in text that is automatically extracted from the information 



automatically generating database queries from said 



derived 
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stream, where the topics correspond to predet e rm i ned topic taxonomies_ofthe 
predetermined taxonomies . 

5. (Original) A method as in claim 1 , wherein the step of examining 
comprises a step of automatically extracting text from the information stream. 

6. (Original) A method as in claim 5, wherein the step of examining further 
comprises steps of: 

segmenting the text into sentences; and 

operating on the sentences to identify topics that correspond to 
predetermined topic taxonomies, wherein the step of automatically generating 
database queries operates on identified topics. 

7. (Original) A method as in claim 5, wherein the step of automatically 
extracting text from the information stream comprises a step of operating a voice 
recognition system. 

8. (Original) A method as in claim 5, wherein the step of automatically 
extracting text from the information stream comprises a step of extracting closed 
caption text. 

9. (Original) A method as in claim 5, wherein the step of automatically 
extracting text from the information stream comprises a step of operating a 
character recognition system. 

10. (Original) A method as in claim 5, wherein the step of automatically 
extracting text from the information stream comprises a step of also generating 
text that is descriptive of a number of human faces that are present in an image 
conveyed by the information stream. 
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1 1 . (Currently amended) A method as in claim 5, wherein the step of 
examining the information stream further comprises steps of segmenting the text 
into sentences and a step of operating on the sentences to identify topics that 
correspond to pr o d o t o rmin e d topic taxonomies of the predetermined taxonomies 
and the presence of names of entities, and further comprising steps of 
assembling a list comprised of an identified topic having a start time and an end 
time, as well as any named entities that occur between the start time and the end 
time, assembling a query object comprised of named entities that occur between 
the start time and the end time of the identified topic, searching at least one 
database to identify a first set of stored documents that correspond to the topic, 
identifying a subset of the first set of documents that contain the named entities, 
identifying a second set of documents that correspond to words found in the text; 
scoring the returned documents based on a plurality of criteria and ranking the 
documents based on their scores. 

12. (Currently amended) A method for providing collateral information for 
multiplexing with an information stream, comprising steps of: 

converting the information stream into text; 

analyzing the text to identify information elements based on one or more 
predetermined taxonomies ; 

automatically generating queries from the information elements for 
searching at least one database; 

extracting data from database search results that is relevant to the 
information stream; and 
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multiplexing the data into the information stream for presentation at a 
destination of the information stream. 

13. (Original) A method as in claim 12, wherein the step of extracting 
comprises a step of ranking the extracted information based on a plurality of 
criteria, and where the step of multiplexing uses the ranked data. 

14. (Currently amended) A method as in claim 12, wherein the step of 
extracting comprises a step of ranking extracted document information based on 
a score derived from a free text search of a document database using the text, 
on a number of named entities extracted from the text that are found in the 
documents, and on a taxonomy path score, where the taxonomy path score 
represents an amount of relatedness between a taxonomy-related information 
element identified in the text and a tree of the predetermined taxonomy 
taxonomies tree. 

15. (Currently amended) A method as in claim 12, wherein the queries are 
generated based on information elements that correspond to a list of information 
elements identifying topics in the text being analyzed, where the topics 
correspond to pr o d o tormin e d t opic taxonomies of the predetermined taxonomies . 

16. (Currently amended) A method as in claim 12, wherein the step of 
analyzing the text comprises steps of segmenting the text into sentences and a 
step of operating on the sentences to identify topics that correspond to 
prodotormined topic taxonomies of the predetermined taxonomies , and wherein 
the step of automatically generating queries operates on identified topics. 

17. (Currently amended) A method as in claim 12, wherein the step of 
analyzing the text comprises steps of at least segmenting the text into sentences, 
identifying names of entities within the text, and a step of operating on the 
sentences to identify topics that correspond to_ pr e d e t e rmin e d topic taxonomiesjof 
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the predetermined taxonomies , and wherein the step of automatically generating 
queries operates on identified topics and ranks the database search results 
based at least on numbers of named entities found and on an amount of 
relatedness between a taxonomy-related information element identified in the 
text and a tree of the predetermined taxonomy taxonomies tfee. 

18. (Currently amended) A system for providing collateral information for 
inclusion with an information stream, said system operating in real time or 
substantially real time and comprising: 

a subsystem for examining the information stream to recognize a 
presence of events that occur in the information stream , wherein said events are 
derived from the information stream based on one or more predetermined 
taxonomies ; 

a subsystem, having an input coupled to an output of said examination 
subsystem, for automatically generating database queries from said r o cogn i z e d 
derived events; 

a database for receiving said database queries; and 

a subsystem, having an input coupled to an output of said database, for 
analyzing results of said database queries so as to rank and select said results to 
be inserted into the information stream as information that is collateral to said 
recogniz e d derived events. 



19. (Currently amended) A system as in claim 18, wherein the analyzing 
subsystem employs ranking criteria comprised of a score derived from a free text 
search of the database using text that is automatically extracted from the 
information stream, on a number of named entities appearing in the text and in 
the database query results, and on a taxonomy path score, where the taxonomy 
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path score represents an amount of relatedness between a taxonomy-related 
information element found in the text and a tree of the predetermined taxonomy 
taxonomiest ree, and wherein the query generation subsystem generates queries 
based on information corresponding to a list that identifies topics in the text that 
is automatically extracted from the information stream, where the topics 
correspond to elements of the taxonomy tree. 

20. (Currently amended) A system as in claim 18, wherein said examining 
subsystem comprises at least one unit for automatically extracting text from the 
information stream, a unit for segmenting the text into sentences and at least one 
unit for operating on the sentences to identify topics that correspond to 
pr o d o termin e d t opic taxonomies of the predetermined taxonomies , wherein said 
query generation subsystem automatically generates database queries based at 
least in part on identified topics. 

21. (Original) A system as in claim 20, wherein said text extracting unit 
comprises at least one of a voice recognition system, a system for extracting 
closed caption text, and a character recognition system. 

22. (Original) A system as in claim 20, wherein said examining subsystem 
comprises a unit for generating text that is descriptive of a number of human 
faces that are present in an image conveyed by the information stream. 

23. (Original) A system as in claim 20, and further comprising a unit for 
operating on the sentences to identify the presence of names of entities, and 
further comprising a unit for assembling a list comprised of an identified topic 
having a start time and an end time, as well as any named entities that occur 
between the start time and the end time, and where the query generation 
subsystem assembles a query object comprised of named entities that occur 
between the start time and the end time of the identified topic for searching said 
database to identify a first set of stored documents that correspond to the topic, a 
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subset of the first set of documents that contain the named entities, a second set 
of documents that correspond to words found in the text; and where said 
analyzing subsystem scores the returned documents based on a plurality of 
criteria and ranks the documents based on their scores. 

24. (Currently amended) A computer-implemented method for generating 
collateral information from an audio/video stream, comprising steps of: 

examining the audio/video stream to recognize a presence of events that 
occur in the audio/video stream , wherein said events are derived from the 
information stream based on one or more predetermined taxonomies ; 

generating database queries from said recogniz e d derived events; and 

analyzing results of said database queries so as to rank and select said 
results to be presented as information that is collateral to said rooogn i zod derived 
events. 

25. (Previously presented) A method as in claim 24, and further 
comprising a step of inserting the collateral information into the audio/video 
stream in real time or substantially real time. 

26. (Previously presented) A method as in claim 24, wherein the step of 
examining includes a step of generating a speech transcript from at least the 
audio portion of the audio/visual stream, and wherein recognized events 
comprise speech topics. 

27. (Previously presented) A method as in claim 24, wherein the 
audio/video stream originates as a television broadcast signal. 
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28. (Previously presented) A method as in claim 24, wherein the 
audio/video stream originates at a meeting, and further comprising a step of 
presenting the collateral information to meeting participants in real time or 
substantially real time. 

29. (Previously presented) A method as in claim 28, wherein the step of 
presenting comprises a step of inserting the collateral information into the 
audio/video stream, and displaying the audio/video stream to the meeting 
participants. 

30. (Previously presented) A method as in claim 28, and further 
comprising a step of archiving at least the collateral information. 

31. (Currently amended) A method as in claim 24, wherein the database 
queries are automatically generated based on information corresponding to 
identified topics extracted from the audio/video stream, where the topics 
correspond to pr e d e term i n e d topic taxonomies of the predetermined taxonomies . 

32. (Currently amended) A method as in claim 24, wherein the step of 
examining includes steps of generating a speech transcript comprised of words 
from at least the audio portion of the audio/video stream; segmenting the words 
into sentences; and operating on the sentences to identify topics that correspond 
to pred e t e rm i n e d topic taxonomies of the predetermined taxonomies , wherein the 
step of generating database queries operates on identified topics. 

33. (Currently amended) A computer readable media having recorded 
thereon a program for providing collateral information for inclusion with an 
information stream, the program comprising instructions for examining the 
information stream to recognize a presence of events that occur in the 
information stream, wherein the events are derived from the information stream 
based on one or more predetermined taxonomies, for automatically generating 
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database queries from r e cognized derived events; and for analyzing database 
query results so as to rank and select database query results to be inserted into 
the information stream as collateral information. 
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